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FOREWORD 

Among  the  responsibilities  assigned  to  the  Office  of  the  Manager,  National 
Communications  System,  is  the  management  of  the  Federal  Telecommunication 
Standards  Program.  Under  this  program,  the  NCS,  with  the  assistance  of  the 
Federal  Telecommunication  Standards  Committee  identifies,  develops,  and 
coordinates  proposed  Federal  Standards  which  either  contribute  to  the 
interoperability  of  functionally  similar  Federal  telecommunication  systems  or 
to  the  achievement  of  a  compatible  and  efficient  interface  between  computer  and 
telecommunication  systems.  In  developing  and  coordinating  these  standards,  a 
considerable  amount  of  effort  is  expended  in  initiating  and  pursuing  joint 
standards  development  efforts  with  appropriate  technical  committees  of  the 
Electronics  Industries  Association,  the  American  National  Standards  Institute, 
the  International  Organization  for  Standardization,  and  the  International 
Telegraph  and  Telephone  Consultative  Committee  of  the  International 
Telecommunication  Union.  This  Technical  Information  Bulletin  presents  an 
overview  of  an  effort  which  is  contributing  to  the  development  of  compatible 
Federal,  national,  and  international  standards  in  the  area  of  Video 
Teleconferencing.  It  has  been  prepared  to  inform  interested  Federal  activities 
of  the  progress  of  these  efforts.  Any  comments,  inputs  or  statements  of 
requirements  which  could  assist  in  the  advancement  of  this  work  are  welcome  and 
should  be  addressed  to: 
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FINAL  REPORT 

DEVELOPMENT  OF  A  VIDEO  TAPE  TO  CORRELATE 
SUBJECTIVE  AND  OBJECTIVE  TESTING  OF  TELECONFERENCE  SYSTEMS 

SECTION  1  -  INTRODUCTION  AND  SUMMARY 

This  document  summarizes  work  performed  by  Delta  Information 
Systems,  Inc.  for  the  National  Communications  System,  Office  of 
Technology  and  Standards.  This  office  is  responsible  for  the 
management  of  the  Federal  Telecommunications  Standards  Program, 
which  develops  telecommunications  standards,  whose  use  is 
mandatory  for  all  Federal  departments  and  agencies.  This  study 
was  performed  under  task  order  number  87-011  of  contract  number 
DCA100-87-C-0078  . 

This  report  covers  the  design  and  production  of  two  video 
tapes  for  testing  full  motion  codecs.  It  is  part  of  a  program 
for  comparative  evaluation  of  motion  codecs  used  for 
teleconferencing  at  a  wide  range  of  bit  rates.  The  specific 
purpose  of  the  subjective  tape  is  to  provide  pictorial  material 
which  may  occur  during  both  government  and  commercial 
teleconferences  and  related  applications  which  simultaneously 
stresses  the  capabilities  of  each  codec  sufficiently  so  that 
differences  in  performance  will  become  apparent.  The  choice  of 
material  must  lead  to  a  completely  impartial  evaluation  without 
favoring  any  particular  algorithm,  and  should  not  be  constrained 
by  limitations  in  encoding  and  transmission. 

The  objective  motion  test  tape  provides  a  means  to  develop  a 
numerical  score  describing  the  motion  performance  of  a  codec 


under  specific  test  conditions.  This  is  a  new  endeavor  which 
required  a  considerable  amount  of  design  and  experimentation 
which  was  performed  as  part  of  Task  No.  87-010.  The  finally- 
produced  tape  is  based  on  the  results  of  these  experiments  and  is 
usable  for  testing  video  codecs  over  the  full  range  of  bit  rates. 

Section  2  of  this  report  covers  the  background  material  on 
which  this  task  was  based  and  gives  an  overview  of  all  the 
interrelated  activities  which  were  performed  to  establish  an 
integrated  program  for  the  evaluation  of  video  teleconf erencing 
codecs.  Details  of  the  test  tapes  which  were  prepared  for 
objective  and  subjective  testing  are  given  in  Sections  3  and  4. 

A  brief  conclusion  and  recommendations  for  future  efforts  are 
contained  in  Section  5. 
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SECTION  2  -  BACKGROUND  AND  OVERVIEW 

This  effort  is  closely  related  to  two  previous  tasks  within 
the  same  overall  program.  They  are  Task  No.  87-007,  Development 
of  a  Video  Tape  to  Test  Video  Codecs  Operating  at  64  Kbps,  and 
Task  No.  87-010,  Standardization  of  End-to-End  Performance  for 
Full  Motion  Video  Teleconferencing.  Both  these  tasks  in  turn  use 
background  material  prepared  in  previous  years  as  parts  of  other 
NCS  programs . 

The  video  test  tape  developed  for  testing  low  bit  rate 
codecs  (NCS  Technical  Information  Bulletin  [TIB]  89-2, 

Development  of  a  Video  Test  Tape  to  Test  Video  Codecs  Operating 
at  64  KBPS  -  Final  Report,  February,  1989)  used  much  previously 
prepared  material.  Scenes  of  highest  technical  quality, 
described  in  TIB  85-2,  Development  of  a  Video  Tape  to  Test 
Teleconferencing  Codecs,  August  23,  1985,  which  had  proved  their 
usefulness  during  previous  subjective  codec  tests  (TIB  85-5  , 

Test  and  Evaluation  of  Teleconferencing  Video  Codecs  Transmitting 
at  1.5  MBPS,  August  23,  1985)  were  incorporated,  including  both 
people  and  graphics.  Additional  material  was  produced,  covering 
both  videophone  scenes  and  graphics.  The  final  tape  is  divided 
into  three  parts,  namely: 

Part  A:  Still  Graphics 
Part  B:  Motion  Graphics 
Part  C:  Limited  Motion 

While  the  contents  of  Part  C  are  deliberately  limited  to  be  used 
for  subjective  testing  of  low  bit  rate  codecs  (below  384  Kbps) , 
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the  graphic  material  is  applicable  to  all  codecs,  regardless  of 
bit  rate.  Therefore,  Parts  A  and  B  will  also  be  used  with  the 
full  motion  test  tape  produced  under  the  present  task  which  is 
designated  Part  D.  The  anticipated  application  of  all  these 
tapes  is  given  on  Table  2-1.  The  bit  rates  given  for  Tapes  C  and 
D  are  typical  examples  and  fit  into  the  subsequently  established 
CCITT  hierarchy  of  P  x  64  Kbps. 

The  purpose  of  the  standardization  task  is  to  establish 
performance  requirements  and  objective  test  methods  for  video 
teleconferencing  systems.  It  requires  adaptation  of  conventional 
analog  tests  for  use  with  digital  systems  and  development  of 
novel  techniques  for  evaluation  of  motion  performance.  Since  the 
use  of  high  quality  video  tapes  has  long  been  the  standard  method 
for  testing  the  performance  of  digital  video  systems  it  was 
attempted  to  record  the  required  test  signals  on  1"  video  tape. 
However,  the  recording  process  caused  too  much  distortion  of  the 
highly  critical  test  signals,  therefore  this  approach  could  not 
be  recommended. 

The  major  effort  of  the  standardization  task  was  the 
development  of  a  methodology  for  objective  testing  of  the  motion 
performance  of  video  codecs.  This  required  a  completely  new 
approach  by  means  of  a  series  of  computer  generated  test  patterns 
with  accurately  controlled  motion.  An  experimental  tape 
containing  such  patterns  was  processed  through  3  different  codecs 
operating  over  the  full  range  of  conventional  bit  rates.  The 
processed  tapes  were  viewed  on  a  waveform  monitor  and  the  thus 
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APPLICATION 


obtained  data  evaluated  to  produce  numerical  results.  These 
results  showed  general  consistency  and  good  correlation  with  the 
various  codec  and  test  pattern  parameters  and  also  with  known 
subjective  evaluations.  This  proved  the  validity  of  the  general 
approach  and  made  it  possible  to  use  the  experimental  tape  as 
source  material  for  a  motion  test  tape  which  can  be  used  to 
establish  correlation  between  subjective  and  objective  codec 
performance  tests. 

The  overall  objective  of  the  present  program  together  with 
the  two  just  described  previous  tasks  is  an  integrated  approach 
to  testing  complete  video  teleconferencing  systems  over  the  full 
range  of  generally  used  bit  rates.  Figure  2-1  schematically 
shows  the  interrelation  of  the  various  steps  taken  in  the 
implementation  of  each  of  the  three  tasks.  The  result  will  be 
performance  and  test  standards  for  all  video  teleconferencing 
systems  and  the  ability  to  develop  performance  specifications  in 
accordance  with  system  requirements. 


2 


4 


Standardization  of  Qid-to-&id 
Performance  for  Full  Motion 
Video  Teleconferencing  (Task  87-10) 


Static  Tests 


I*  Select  Applicable  Broadcast  Performance 
Parameters  &  Analog  Test  Signals 

*  Review  &  Adapt  Previously  Suggested 
Performance  Standards 


Motion  Tests 


*  Design  &  Produce  Artificial  Test  Pattern 

*  Process  Experimental  Test  Tape  Through 
Codecs  at  Various  Bit  Rates 


*  Analyze  Processed  Tapes  to  Establish  Test 
Methodology  &  Parameters 


Development  of  a  Video  Tape  to  Test  Video 
Codecs  Operating  at  64  Kbps  (Task  87-7) 


*  Select  Graphic  and  Limited  Motion 
Scenes  From  Existing  Tapes 

*  Produce  Tape  of  Videophone  Scenes 
Select  Most  Typical  Scenes 

*  Produce  Tape  Containing  New  Still 
Graphic  and  Motion  Graphic  Scales 

*  Edit  the  Following  Test  Tapes: 

A:  Still  Graphics 
B:  Motion  Graphics 
C:  Limited  Motion 


Development  of  a  Video  Tape  to  Correlate 
Subjective  and  Objective  Testing  of 
Teleconferencing  Systems  (Task  87-11) 


Objective  Tape 


*  Analog  Test  Signals  Not  Included 
Because  Quality  Not  Satisfactory. 
Tests  to  Be  Performed  Directly 
With  Signal  Generator. 


*  Select  most  Suitable  Test  Patterns 
&  Edit  Final  Motion  Test  Tape 


Subjective  Tape 


*  Select  Full  Motion  Scenes  From 
Basting  Tapes 

*  Select  Full  Motion  Videophone  Scenes 

*  Edit  Test  Tape  D:  Full  Motion 

*  Also  Use  Previously  Prepared 
Test  Tapes  A  &  B 


FIGURE  2-1  PROGRAM  CVJRVnV 


SECTION  3 


OBJECTIVE  TEST  TAPE 


3.1  Still  Picture  Analog  Tests 

Analog  performance  standards  covering  mostly  the  same 
parameters  as  conventional  broadcast  performance  specifications 
have  been  developed  previously  in  Task  87-010  and  are  shown  in 
the  subject  final  report  on  Table  3-3.  The  applicability  and 
numerical  specification  for  each  parameter  still  have  to  be 
verified  by  system  tests.  The  original  intent  of  this  program 
was  to  record  the  required  analog  test  signals  on  video  tape  and 
thus  produce  an  integrated  objective  codec  test  tape.  However, 
previous  tests  have  shown  that  the  taping  process  itself  produces 
distortions  which  are  significant  enough  to  make  it  impossible  to 
obtain  valid  performance  data. 

The  only  practical  way  to  obtain  dependable  results  is  to 
have  test  equipment  directly  at  the  teleconferencing  system 
terminals.  Not  long  ago  that  could  have  been  quite  awkward  but 
modern  high  performance  compact  equipment  make  such  an 
arrangement  practical.  The  Tektronix  1910  Test  Signal  Generator 
and  VM-700  Video  Measurement  Set  (both  available  at  Delta 
Information  Systems)  are  ideally  suited  for  the  purpose.  They 
provide  a  large  variety  of  digitally  generated  test  signals  and 
both  waveform  and  vector  displays  for  precision  measurements  in 
addition  to  the  capability  of  direct  numerical  printout  of 
selected  lists  of  performance  parameters.  These  equipments 
satisfy  present  and  future  test  requirements  of  video 
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teleconferencing  systems. 

3 . 2  Motion  Tests 

The  experimental  motion  test  tape  which  was  prepared  as  part 
of  Task  87-010  contained  two  different  types  of  artificial 
motion.  The  simplest  approach  uses  a  scene  cut  by  switching 
between  a  pattern  of  white  circles  and  a  black  field  as  shown  in 
Figure  3-1.  The  more  complex  pattern,  a  sample  of  which  is  shown 
in  Figure  3-2,  provides  actual  motion  implemented  by  black  spokes 
rotating  over  a  background  consisting  of  white  and  colored  areas. 
Both  patterns  produce  an  exactly  definable  and  easily  calculated 
amount  of  pixel  change  per  frame.  This  amount  can  be  changed  by 
varying  the  detail  in  the  picture  (such  as  size  and  spacing  of 
circles  or  number  and  width  of  wheel  spokes)  and  by  changing  the 
speed  of  switching  or  rotation.  These  two  types  of  pixel  change 
can  be  combined  into  the  common  parameter  of  temporal  frequency. 

Extensive  tests  with  the  experimental  tape  showed  that  both 
methods  of  producing  artificial  motion  were  usable.  The  rotating 
wheel  yielded  somewhat  more  consistent  results.  Furthermore,  it 
may  be  possible  to  design  a  codec  algorithm  which  responds  well 
to  a  scene  cut  without  having  good  performance  with  actual 
motion.  Therefore,  it  was  decided  that  using  only  the  rotating 
wheel  pattern  was  preferable  and  sufficient  to  test  codec  motion 
performance . 

Originally,  three  spoke  widths  were  arbitrarily  selected 
to  produce  pictures  with  different  amounts  of  detail;  they  were 
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ROTATING  OVERLAY 


FIGURE  3-2: 


30  DEGREE  PATTERN 


30,  18,  and  10  degrees.  The  tests  indicated  that  this  choice  was 
well  made  to  cover  the  full  range  of  codec  bit  rates.  Testing 
1.544  Mbps  codecs  required  the  use  of  the  10  degree  spokes  to 
sufficiently  stress  codec  performance  while  at  64  Kbps  mainly  30 
degree  spokes  produced  a  temporal  response  curve  with  a 
significant  range  and  slope.  The  18  degree  spokes  were  most 
desirable  for  testing  384  Kbps  codecs.  Using  each  pattern  over 
the  implemented  range  of  rotation  speeds  resulted  in  one  temporal 
response  curve . 

The  preparation  of  the  test  tape  was  subject  to  some 
constraints  imposed  by  the  CUBICOMP  computer  animation  equipment 
in  use.  The  normal  time  increment  for  a  picture  change  is  one 
frame  and  rotation  increments  must  be  in  full  degrees.  These 
factors  can  seriously  limit  the  obtainable  rotation  speeds  of  the 
pattern  but  it  is  possible  to  use  holding  times  of  more  than  one 
frame  and  alternate  both  holding  times  and  rotation  increments 
between  consecutive  frames.  This  way  a  sufficient  variety  of 
rotation  speeds  can  be  produced. 

Table  3-1  lists  the  patterns  that  have  been  implemented  on 
the  objective  motion  test  tape.  Following  the  spoke  width  it 
gives  the  rotation  speed  in  two  different  terms.  For 
compatibility  with  the  CUBICOMP  equipment  it  was  first  specified 
in  frames  per  revolution  but  from  the  user’s  point  of  view 
degrees  per  second  is  much  more  descriptive.  The  following 
parameters  are  readily  derived  arithmetically  from  spoke  width 
and  rotation  speed.  Temporal  frequency  is  the  most  useful 
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parameter  to  describe  the  motion  response  of  a  codec.  The  number 
of  frames  per  spoke  gives  the  period  required  for  one  spoke  to 
pass  a  reference  point  in  the  picture.  This  is  important  in  the 
case  of  low  bit  rate  codecs  which  often  use  a  high  rate  of 
vertical  subsampling.  Tf  the  number  of  frames  per  spoke 
approaches  and  finally  becomes  lower  than  the  rate  of  vertical 
subsampling  some  spokes  (both  black  and  white)  will  be  severely 
distorted  and  may  actually  be  skipped  completely.  This  leads  to 
an  effect  which  may  be  called  temporal  aliasing  which  produces 
false  results.  The  percentage  of  pixel  change  per  frame 
(computed  as  the  percentage  of  the  total  area  covered  by  the 
moving  spoke  during  one  frame)  is  a  good  measure  of  the  stress 
imposed  on  the  motion  rendition  capability  of  a  codec.  However, 
in  the  very  common  case  of  codecs  using  transform  coding  which 
divides  the  picture  into  blocks  the  percentage  of  block  change 
per  spoke  is  even  more  descriptive.  This  number  is  difficult  to 
compute  accurately  but  an  approximate  graphic  method  was  used  to 
come  up  with  a  sufficient  sample  of  values.  The  last  column  of 
Table  3-1  shows  a  wide  range  of  values  with  only  little  overlap 
between  the  three  spoke  widths.  This  is  quite  different  from 
other  parameters  and  also  confirms  the  experimental  result  that 
all  3  implemented  spoke  widths  are  needed  for  full  coverage. 

There  would  be  no  overlap  of  the  block  change  percentage  numbers 
if  only  30  and  10  degree  spokes  were  used. 

On  the  edited  motion  test  tape  the  time  for  each  pattern  was 
held  to  20  seconds.  This  is  shorter  than  on  the  original 
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experimental  tape  but  based  on  early  test  results  was  deemed 
sufficient.  The  performed  tests  have  shown  that  at  low  temporal 
frequencies  only  about  4  temporal  cycles  have  to  be  examined  to 
produce  dependable  results.  The  lowest  temporal  frequency  on  the 
tape  is  0.33  cps  which  means  that  12  seconds  are  required  for  4 
cycles.  Therefore,  a  duration  of  20  seconds  is  more  than 
adequate.  Even  though  at  higher  temporal  frequencies  it  is 
desirable  to  examine  more  cycles,  the  measurement  can  be 
performed  in  a  shorter  time  but  for  the  purpose  of  uniformity  the 
duration  of  20  seconds  was  maintained  for  all  test  patterns. 
Preceding  each  pattern  there  is  a  title  with  the  pertinent 
parameters  which  is  held  for  6  seconds.  Including  an 
introductory  title  results  in  a  total  tape  length  of  just  over  10 
minutes . 
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SECTION  4  -  SUBJECTIVE  TEST  TAPE 

The  large  amount  of  already  available  test  scenes  made  the 
preparation  of  new  material  unnecessary.  Only  selection  of 
previously  taped  scenes  and  editing  was  needed  to  produce  the 
final  tape. 

This  subjective  test  tape  contains  only  full  motion  scenes 
with  people  and  is  expected  to  be  used  at  codec  bit  rates  of  384 
Kbps  and  above.  It  is  designated  Part  D  of  a  series  of 
subjective  test  tapes  which  together  are  to  be  usable  for  all 
existing  and  future  codecs.  Part  A,  Still  Graphics,  and  Part  B, 
Motion  Graphics,  though  previously  prepared  for  use  with  low  bit 
rate  codecs,  are  equally  usable  in  the  higher  bit  rate  range. 
Still  graphics  test  both  resolution  and  color  rendition 
capability  which  are  important  for  all  codecs  and  are  designed  to 
be  challenging  in  both  freeze  frame  and  motion  modes.  Motion 
graphics  add  the  requirement  to  reproduce  moving  material, 
pointing  and  drawing,  and  camera  zooming.  These  features  stress 
codec  capability  at  all  bit  rates,  only  settling  time  after 
motion  becomes  shorter  and  therefore  less  important  at  higher 
rates . 

The  scenerio  of  the  full  motion  subjective  test  tape, 
designated  Part  D,  is  shown  on  Table  4-1.  It  consists  of  18 
sequences,  separated  by  10  seconds  long  scoring  intervals. 
Including  the  introductory  title  results  in  a  total  tape  length 
of  about  15  minutes. 

The  highest  quality  scenes  were  selected  from  previously 
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TAPE  SCENARIO 
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ISACOMM  Girl  at  Chalk  Board 


produced  tapes  which  had  been  proven  to  give  good  results  in 
subjective  evaluations.  Most  material  was  prepared  at  Center 
City  Video  (CCV)  in  Philadelphia  but  some  scenes  which  were 
produced  in  the  former  ISACOMM  (now  US  Sprint)  Teleconference 
Room  have  been  added.  Several  sequences  consist  of  scenes  which 
are  typical  for  videophone.  These  sequences  also  include  sound 
which  may  be  important  in  the  evaluation  of  some  codecs.  A  few 
scenes  which  were  judged  to  be  useful  to  test  some  applications 
of  codecs  at  all  bit  rates  were  repeated  from  the  Limited  Motion 
Test  Tape  C  but  occasionally  with  slightly  modified  editing.  The 
resulting  tape  consists  of  scenes  depicting  the  variety  of 
applications  of  video  teleconferencing  codecs  requiring  good 
rendition  of  full  motion. 
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SECTION  5  -  CONCLUSION  AND  RECOMMENDATIONS 

The  output  of  this  task  consists  of  the  two  test  tapes 
decribed  in  Sections  3  and  4.  Both  tapes  were  originally 
produced  as  1"  tapes.  The  objective  motion  tests  do  not  require 
the  highest  resolution  and  color  rendition  capability,  therefore 
it  is  recommended  and  expected  that  only  3/4"  copies  will  be  used 
for  actual  tests.  The  subjective  full  motion  test  tape  is 
available  in  both  tape  sizes.  For  highest  performance  and 
precision  1"  tape  is  very  desirable  but,  if  equipment 
availability  and/or  cost  make  this  impractical,  the  use  of  modern 
high  performance  recorders  makes  3/4”  tape  fully  acceptable. 

The  next  step  in  the  development  of  codec  testing  is  the  use 
of  the  just  produced  tapes,  together  with  the  previously 
described  tapes  A  and  B  for  graphics  (see  Task  87-7)  for 
correlation  of  subjective  and  objective  test  results  on  codecs 
operating  at  384  Kbps  and  above.  The  same  should  be  done  on 
codecs  operating  below  384  Kbps  -  mainly  at  64  Kbps  -  using  tape 
C  instead  of  tape  D.  The  results  of  these  tests  are  expected  to 
confirm  the  validity  of  objective  motion  testing  using  the 
rotating  wheel  pattern. 

Objective  codec  tests  will  be  made  using  the  just  developed 
motion  test  tape  for  motion  performance  and  standard  analog  test 
equipment  for  still  picture  performance.  The  results  of  these 
tests  will  indicate  the  degree  of  correlation  between  subjective 
and  objective  testing.  They  will  also  show  the  relevance  of  the 
various  analog  perfomance  parameters  and  probably  lead  to  a 


reduction  of  the  number  to  be  tested.  More  experience  with  the 
rotating  wheel  patterns  may  result  in  modifications  of  the 
pattern  design  and  the  required  range  of  spoke  widths,  rotation 
speeds  and  temporal  frequencies.  The  ultimate  goal  is 
elimination  of  the  motion  test  tape  by  replacing  it  with  a  motion 
pattern  generator  and  automatic  analyzer.  However,  this  can  be 
achieved  only  after  extensive  use  of,  and  experience  with,  the 
test  material  produced  within  the  present  task. 
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